Bard College

Bard Digital Commons

Senior Projects Spring 2015 Bard Undergraduate Senior Projects

2015

Development and Optimization of a Two-View

Model for Anamorphic Projections on Planar

Surfaces

Van Mai Nguyen "i

Bard College

8is On-Campus only is brought to you for free and open access by the

Bard Undergraduate Senior Projects at Bard Digital Commons. It has been

accepted for inclusion in Senior Projects Spring 2015 by an authorized

administrator of Bard Digital Commons. For more information, please

contact digitalcommons@bard.edu.

Recommended Citation

Nguyen 8i, Van Mai, "Development and Optimization of a Two-View Model for Anamorphic Projections on Planar Surfaces" (2015).

Senior Projects Spring 2015. Paper 306.

h9p://digitalcommons.bard.edu/senproj_s2015/306

Development and Optimization of a

Two-View Model for Anamorphic

Projections on Planar Surfaces

A Senior Project submitted to

The Division of Science, Mathematics, and Computing

Bard College

Van Mai Nguyen Thi

Annandale-on-Hudson, New York

May, 2015

Abstract

An anamorphic projection is an image that is intentionally distorted so that the original

image can be seen only when looked at from a certain perspective, or using a special device,

for example a mirror. Origins of anamorphosis can be traced back to the 16th century

art, but beyond the aesthetic values, anamorphosis has found its uses in many practical

settings, such as road signs, and keystone correction. Despite the widespread uses, there is

a scarcity of detailed explanation of the mathematical principles behind anamorphosis, and

there is hardly any computer software generating the anamorphic projections. The goal of

this project is to apply computer vision techniques to create a program that automates the

process of generating anamorphic projections. We use a projector-camera system to derive

homography mappings between the projector and camera image, and propose a method

for generating an optimal anamorphic image for multiple viewers using least squares.

Contents

Abstract 1

Dedication 6

Acknowledgments 7

1 Introduction 8

1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

1.2 Previous Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1.3 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.4 Summary of Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2 Mathematical Concepts 13

2.1 Pinhole Camera Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.1.1 Mathematical Description . . . . . . . . . . . . . . . . . . . . . . . . 14

2.2 Homogeneous Representation . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.3 2D Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.3.1 Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.3.2 Warping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3 Algorithmic Methods 24

3.1 Least Squares Minimization Method . . . . . . . . . . . . . . . . . . . . . . 24

3.1.1 Linear Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

3.1.2 General Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3.2 Projective Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.2.1 Homography Estimation with Linear Least Squares . . . . . . . . . . 31

3.2.2 Homography Estimation with Homogeneous Linear Least Squares . 33

Contents 3

3.3 Feature Detection Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.3.1 Harris Corner Detection . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.3.2 Good Features To Track . . . . . . . . . . . . . . . . . . . . . . . . . 36

4 Results 37

4.1 Basic Planar Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4.1.1 Pattern Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

4.1.2 Homography Estimation . . . . . . . . . . . . . . . . . . . . . . . . . 41

4.1.3 Generating the Anamorphic Image . . . . . . . . . . . . . . . . . . . 41

4.1.4 Challenges and Limitations . . . . . . . . . . . . . . . . . . . . . . . 43

4.2 Improved Planar Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

4.2.1 Modiﬁed Set-up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

4.2.2 Detecting the Projection Screen . . . . . . . . . . . . . . . . . . . . . 47

4.2.3 Homography Estimation with Chessboard Corner Detection . . . . . 48

4.2.4 Repositioning and Resizing the Pre-anamorph . . . . . . . . . . . . . 49

4.2.5 Challenges and Limitations . . . . . . . . . . . . . . . . . . . . . . . 51

4.3 Two-view Anamorphosis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

4.3.1 Two-view Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

4.3.2 Target Camera Images . . . . . . . . . . . . . . . . . . . . . . . . . . 55

4.3.3 Homography Estimation . . . . . . . . . . . . . . . . . . . . . . . . . 57

4.3.4 Generating the Anamorphic Image . . . . . . . . . . . . . . . . . . . 62

5 Future Work 64

Bibliography 66

List of Figures

1.1.1 Hans Holbein’s The Ambassadors, 1533 . . . . . . . . . . . . . . . . . . . . 9

1.1.2 Undistorted skull from H. Holbein’s The Ambassadors . . . . . . . . . . . . 9

1.1.3 Road sign in the shape of an elongated bike. Left: view from the top; right:

view from a driver’s perspective [7]. . . . . . . . . . . . . . . . . . . . . . . . 10

2.1.1 Graphical representation of the pinhole camera model. [3] . . . . . . . . . . 14

2.3.1 Examples of basic 2D transformations. [13] . . . . . . . . . . . . . . . . . . 19

3.2.1 The homographies between the projector image, camera image, and projec-

tion surface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4.1.1 Steps involved in ﬁnding the homography H

P C

: (a) project a rectangular

pattern (b) Detect the pattern in the camera image . . . . . . . . . . . . . . 39

4.1.2 Verify that the anamorph is seen correctly by the camera: (a) project the

anamorph (b) The image (white rectangle) is seen correctly in the camera

image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4.1.3 Screen shots and camera captures of the program . . . . . . . . . . . . . . . 40

4.1.4 Diagram of the scenario in which the points in the camera image are mapped

outside of the projector image: (a) camera image (b) projector image . . . . 42

4.1.5 (a) Anamorphic image not ﬁtting inside the projector image. (b) When the

anamorphic image in (a) is projected, the camera receives an incomplete

image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

4.1.6 Incorrectly detected projection corners. . . . . . . . . . . . . . . . . . . . . 44

4.2.1 Screenshots of the program: (a) Detected chessboard corners. (b) Prepared

pre-anamorph. (c) Anamorphic image after warping. (d) Anamorphic image

seen correctly by the camera. . . . . . . . . . . . . . . . . . . . . . . . . . . 45

4.2.2 Detected projection screen corners marked with blue dots. . . . . . . . . . . 47

LIST OF FIGURES 5

4.2.3 (a) Original chessboard corners in the projector image. (b) Detected chess-

board corners in the camera image. . . . . . . . . . . . . . . . . . . . . . . . 48

4.2.4 Screenshots of the program . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

4.2.5 A scenario in which the program fails to detect the chessboard corners

correctly. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

4.3.1 The new setup: (a) Camera 1 (b) Projector (c) Camera 2 (d) Projection

surface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

4.3.2 The new setup and notation for the homographies. . . . . . . . . . . . . . . 55

4.3.3 New chessboard patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

Dedication

I dedicate this work to my family.

Acknowledgments

First of all, I would like to thank my advisors, Keith and Jim, for their constant advice,

support and encouragement throughout this project. In would like to thank Keith for

his enthusiasm and support, from brainstorming ideas to ﬁnalizing this project. Thank

you Jim for your guidance and contribution. My thanks also go to the Mathematics and

Computer Science departments for giving me the foundation to work on this project.

I also want to thank my friends, Thinh, Linh, Jin, for supporting me with my project

and the good memories that we have had at Bard. I also thank Georgia, Alexzandra and

Eva-Marie for being great suitemates and helping me whenever I needed.

I want to thank Thant Ko Ko for his continuous support and always being there for me.

Finally, I would like to thank my parents for their unconditional love and support

throughout my life, and my little sister Ola for never failing to make me laugh.

Introduction

1.1 Background

An anamorphosis (or anamorphic projection/image) is an image that is intentionally dis-

torted so that the original image can be recovered only when looked at from a certain

point of view or using a special device, for example a mirror. Origins of anamorphosis

can be traced back to the 16th century when artists, mathematicians and philosophers,

fascinated by the idea of perspective, experimented with the notions of illusion, truth and

reality [7]. Some of the notable examples of anamorphosis include Jean-Francois Niceron’s

methods for geometrical construction that generated multiple types of anamorphic trans-

forms which involved both exact and approximate methods [7, 8]. One of the most famous

anamorphic paintings is Hans Holbein’s The Ambassadors (Figure 1.1.1), in which the

artist embedded an anamorphic image of a skull (Figure 1.1.2) at the bottom of the paint-

ing as a memento mori, a recurring motif in many artworks in the Renaissance period.

1. INTRODUCTION 9

Figure 1.1.1: Hans Holbein’s The Ambassadors, 1533

Figure 1.1.2: Undistorted skull from H. Holbein’s The Ambassadors

Anamorphosis is still prevalent to this day, and is incorporated in many new forms of

art, most notably street art. The undying fascination for anamorphosis is manifested in

numerous artworks centered around the anamorphic principle and using it to create a

unique piece of art. Beyond the aesthetic values, anamorphosis has found its uses in many

practical settings, such as road signs that are elongated so that drivers’ looking at them

from a small angle above the ground can see the signs correctly (Figure 1.1.3), or inverted

“ambulance” signs that are designed to be reﬂected correctly in the rear-view mirrors.

1. INTRODUCTION 10

Principles similar to those of anamorphosis are also found in keystone correction, which is

a method for eliminating some distortions of projection that may result from misaligning

the projector with the projection screen.

Despite the widespread uses, there is a scarcity of detailed explanation of the mathe-

matical principles behind anamorphosis in the context of computer vision, and even with

the advanced technology there is hardly any computer software generating the anamorphic

projections. The goal of this project is two-fold. First of all, we explore the mathematical

principles behind the most common types of anamorphosis. Next, we combine these the-

ories with computer vision techniques to create a program that automates the process of

generating anamorphic projections.

Figure 1.1.3: Road sign in the shape of an elongated bike. Left: view from the top; right:

view from a driver’s perspective [7].

1.2 Previous Work

We build our research on previous works on the topics related to anamorphosis. The math-

ematical transformations involved in various types of anamorphosis are explained in [7]. In

particular, it describes the transformation of points between planar surfaces in perspective

anamorphosis, which is the main focus of this project. Because of the lack of resources

1. INTRODUCTION 11

about anamorphosis in computer vision, we review other related research projects that

employ methods that are applicable to anamorphosis. There have been some develop-

ment in the area of keystone correction, a topic which greatly overlaps with our project

as it uses a similar projector-camera setup and planar image transformation detection

methods [2, 12]. Besides keystone correction, further innovations of [2] and [12] include an

interactive user experience through a laser pointer. Related but outside of the scope of our

project is dynamic anamorphosis, which adapts the generated images to the position of

the viewer by tracking the viewer in real-time [9, 11]. We refer to [13] and [6] for compre-

hensive explanations of fundamental computer vision techniques and methods to analyze

images, including feature detection and image transformation. [6] also provides a complete

description of the geometric principles behind multiple view systems and techniques for

reconstructing a three-dimensional scene using projector camera systems.

1.3 Motivation

While the inspiration for this project originates from the intriguing use of anamorphosis

in art, the motivation for our research is the scarcity of previous work exploring the

broad applications of anamorphosis in computer vision. Through this project, we hope to

contribute to and bring the attention to this overlooked subject.

Most research on the related topics deals with reconstructing the three-dimensional ge-

ometry of a scene, for example using a Kinect device or a multiple camera system. This

project, however, takes a diﬀerent approach in that it not only uses the cameras to recon-

struct the information about the scene, but also optimizes the projections for the camera

views. This work focuses on developing anamorphic models for single and two view geome-

tries, both of which have potential applications in visual and performance arts and can

be used to enhance the viewers’ experience and create unique and engaging visual eﬀects.

We employ anamorphic principles to optimize projected images for a multiple view model,

1. INTRODUCTION 12

which has further use in developing anamorphic imaging systems for a large theatrical

audience. Another advantage to this simple projector-camera model is its relatively low

cost, which means that it can be eﬀectively incorporated into class curricula to facilitate

computer vision learning.

1.4 Summary of Results

The results of this project are described in detail in Chapter 4. The ﬁndings in this work are

built upon a review of the mathematical concepts pertinent to perspective anamorphosis

and computer vision techniques, which are employed later in this project.

Chapter 2 discusses the mathematical principles of the Pinhole Camera Model and

two-dimensional image transformations, and introduces the concept of homogeneous coor-

dinates. The theoretical background explained in Chapter 2 serves as a basis for computer

vision applications and methods described in Chapter 3. Based on what we learn from the

review of these concepts, we build three models for generating anamorphic projections on

planar surfaces, discussed in Chapter 4. We ﬁrst implement a basic planar model using

a simple projector-camera system. We estimate the homography relations between the

images in the system and generate anamorphic images for the camera view. Based on

our observations of some limitations in this simple model, we develop an improved model

which addresses some of these limitations. These two models are presented in Sections 4.1

and 4.2, and a homography estimation method using linear least squares for single view

models is explained in Subsection 3.2.1. Finally, Section 4.3 studies a two view model and

Subsection 3.2.2 develops a new method of homography estimation using homogeneous

least squares which we implement in Python as part of the algorithm for the two view

model.

Mathematical Concepts

This chapter deﬁnes some basic concepts and terminology needed to describe anamor-

phic projections. In particular, we present the pinhole camera model and introduce ho-

mogeneous representations of geometrical objects used in computer vision to facilitate

performing image transformations.

2.1 Pinhole Camera Model

The pinhole camera model is one of the simplest camera models. Its principles were known

in as early as the thirteenth century, and used in “camera obscura.” [3]

The pinhole camera can be visualized as a closed box with a small hole on one side, and

an image plane on the opposite side. The light rays, which are reﬂected from objects in

the real world, enter the box through the pinhole, and land on the image plane, forming

a 180

◦

rotated two-dimensional reﬂection, as shown in Figure 2.1.1. The same mechanism

describes the process of human vision, in which the light passes the through the pupil of

the eye.

2. MATHEMATICAL CONCEPTS 14

Figure 2.1.1: Graphical representation of the pinhole camera model. [3]

2.1.1 Mathematical Description

For a more rigorous explanation of the model, we introduce a three-dimensional coordinate

system where Z is the axis orthogonal to the pinhole and image plane, X is the horizontal

axis, and Y is the vertical axis, and the origin is placed at the pinhole, as shown in

Figure 2.1.1. The position of any point in the real world in front of the pinhole box can be

represented with the X, Y and Z coordinates. The image plane has its own two-dimensional

coordinate system with x and y axes parallel to the X and Y axes respectively, and the

origin on the Z-axis at a distance f (focal length) from the pinhole.

Using this model, we can easily predict where a point lands on the image plane given its

three-dimensional position in the real world. Observe that the ray connecting the point in

the real world and its reﬂection in the image plane forms two similar triangles on either side

of the pinhole by projecting itself onto the XZ-plane. Given this observation, we deﬁne

the position (x, y) of a point on the image plane corresponding to the point (X, Y, Z) in

the real world as

y = −f

, and x = −f

2. MATHEMATICAL CONCEPTS 15

2.2 Homogeneous Representation

Homogeneous coordinates were ﬁrst introduced by August Ferdinand M¨obius, and later be-

came a powerful tool with applications in computer vision and projective geometry [1,10].

The homogeneous system is an alternative to the standard Euclidean, or inhomogeneous

system as it solves some of its limitations. One limitation of the Euclidean system is that

it allows only linear transformations that ﬁx the origin to be represented as matrices.

Another limitation is that it does not have a ﬁnite representation for a point or line at

inﬁnity. The homogeneous model solves this problem by introducing an extra dimension,

which provides a convenient representation for points and lines at inﬁnity, as described

later in this section. Another beneﬁt of using the homogeneous model, and the reason for

its prevalence in computer vision, is that projective transformations can be represented as

matrices, which oﬀers a convenient way to perform such operations.

Deﬁnition 2.2.1. Let P be a ﬁnite point in n dimensions, and (x

, x

, ..., x

) be its

representation in Cartesian coordinates. A homogeneous representation of point P is any

(n + 1)-tuple (ωx

, ωx

, ..., ωx

, ω), where ω 6= 0. The last coordinate ω is called the

homogeneous coordinate. A homogeneous representation of a ﬁnite point has a non-zero

homogeneous coordinate, and a point at inﬁnity has a zero homogeneous coordinate.

The homogeneity of this representation lies in the fact that when a homogeneous point

is multiplied by a scalar, it is still the same point and corresponds to the same Cartesian

point.

A Cartesian point can be converted to a homogeneous representation by appending

ω = 1 as the (n + 1)

coordinate. A homogeneous point can be converted to a Cartesian

point by dividing each of its coordinates by ω, and removing the homogeneous coordinate.

2. MATHEMATICAL CONCEPTS 16

Example 2.2.2. Let P

be a two dimensional point with Cartesian coordinates (x, y). A

possible homogeneous representation of P

is (x, y, 1), or more generally (xω, yω, ω) for

any ω 6= 0. Let P

be another point in two dimensions with homogeneous representation

(x, y, ω). P

can be represented in Cartesian coordinates as





Deﬁnition 2.2.3. Let l be a straight line in two dimensions deﬁned by the equation

ax + by + c = 0. Then (a, b, c) is the homogeneous representation of line l.

The homogeneity is preserved in this line representation because (λa, λb, λc) represents

the line λax + λby + λc = 0, which is the same as line ax + by + c = 0 represented by

(a, b, c), for any λ 6= 0.

Additionally, it follows from the deﬁnition of a line that a point P lies on a line l if the

dot product of the homogeneous representations of P and l are zero.

Lemma 2.2.4. A point P is the intersection of two-dimensional lines l

and l

if and

only if P is the cross product of the homogeneous vector representations of l

and l

Proof. This proof consists of two parts. First, we show that the cross product of the

homogeneous representations of two lines is the intersection point of those lines. Then we

prove that an intersection of two lines must also be the cross product of the two lines.

Let l

and l

be two-dimensional non-parallel lines represented as (a

, b

, c

) and

, b

, c

) respectively. Their cross product is a point P deﬁned as

P = (a

, b

, c

) × (a

, b

, c

) = (b

− b

, a

− a

, a

− a

). (2.2.1)

We verify that P lies on both lines l

and l

by demonstrating that the dot product of P

with each line is zero. For line l

we obtain

− b

) + b

− a

) + c

− a

)

= a

− a

+ a

− a

+ a

− a

= 0,

2. MATHEMATICAL CONCEPTS 17

which means that P lies on l

. Similarly, we obtain that P lies on l

− b

) + b

− a

) + c

− a

)

= a

− a

+ a

− a

+ a

− a

= 0.

Since P lies on both l

and l

, P , which is the cross product of the two lines, is the

intersection of those lines.

It remains to show that the intersection of two lines is also the cross product of the

lines. Let Q = (x, y, 1) be the intersection point of l

and l

. Notice that without loss

of generality, any homogeneous representation can be transformed to a form with the

homogeneous coordinate being 1. Because Q is the intersection point, ot must lie on both

lines, which means that the dot product of Q with each line is zero. The dot product of

Q and l

Q · l

= (x, y, 1) · (a

, b

, c

) = a

x + b

y + c

and the dot product of Q and l

Q · l

= (x, y, 1) · (a

, b

, c

) = a

x + b

y + c

We need to solve the following system of equations for x and y:

x + b

y + c

= 0

x + b

y + c

= 0.

We obtain the solution with a simple row reduction:



−c



∼



0 a

− a

−c



2. MATHEMATICAL CONCEPTS 18

∼

0 1

−a

0 −c

−

−a

∼

0 1

−a

1 0

−b

−a

Therefore the coordinates of Q are



− b

− a

, 1



which in homogeneous representation is equivalent to (b

− b

, a

− a

, a

− a

which is the cross product of l

and l

, as computed in the equation (2.2.1). Thus, the

intersection point of two lines is also the cross product of the two lines.

Deﬁnition 2.2.5. The degrees of freedom of a system is the number of independent

parameters that deﬁne the system and are free to vary.

Example 2.2.6. It is clear that a Cartesian point in two dimensions is deﬁned by two

parameters: the x and y coordinate. Thus a 2D point has two degrees of freedom. It is

less obvious that a line in two dimensions has two degrees of freedom. Recall that a line

is deﬁned by an equation of the form ax + by = c, where the coeﬃcients a, b and c deﬁne

the line. Although this equation has three coeﬃcients, a line is unique only up to a scale

of the coeﬃcients, that is a line ax + by = c is the same as a line sax + sby = sc for

any non-zero scalar s. In fact, a line can be deﬁned with two independent parameters: the

slope and the y-intercept, marked as m and c in this common notation: y = mx + c.

2.3 2D Transformations

This section deals with types graphical transformations on 2D images. In computer vision,

these operations are functions of pixel coordinates that return new coordinates for the

given pixel. The simplest types of transformations include translation, rotation and aﬃne.

They form a basis for the projective transformation (homography), which could be thought

2. MATHEMATICAL CONCEPTS 19

of as an advanced transformation that includes but is not limited to a combination of

those basic transformations. The projective transformation is deﬁned in this chapter and

discussed in more detail in Section 3.2. The procedure of applying a transformation to

an image is referred to as warping. There are two main methods of warping, deﬁned in

Subsection 2.3.2.

2.3.1 Transformations

This section focuses on some fundamental transformations of two dimensional images,

illustrated in Figure 2.3.1. For each transformation, we provide a deﬁnition and present

a matrix notation that oﬀers a convenient way to apply them to 2D images. Since the

transformations discussed here do not preserve the origin, their matrix notation is only

possible when we use homogeneous coordinates, introduced in Section 2.2.

Figure 2.3.1: Examples of basic 2D transformations. [13]

Deﬁnition 2.3.1. Translation is a type of transformation that shifts an image by a vector.

It is typically deﬁned as a 2 × 3 matrix

T =



1 0 t

0 1 t



or as a 3 × 3 matrix

T =





1 0 t

0 1 t

0 0 1





for some translation factors t

and t

2. MATHEMATICAL CONCEPTS 20

The 2 × 3 matrix can be applied to a homogeneous point this way:







1 0 t

0 1 t













x + t

y + t



where (x, y, 1) is initial position, and (x

, y

)

is the new position after translation. Observe

that the result of translation using the 2 × 3 matrix is a point in Euclidean coordinates,

without the homogeneous coordinate. This creates a signiﬁcant disadvantage as it prevents

combining several transformations. For example, if we want to apply another translation

on this result, we need to convert it to the homogeneous coordinates ﬁrst.

The 3 × 3 matrix gives the same result, but in homogeneous coordinates:













1 0 t

0 1 t

0 0 1

















x + t

y + t





This notation makes it easy to apply another transformation such as translation directly

to the result, as the result is already in the homogeneous form.

Deﬁnition 2.3.2. Rotation is a type of transformation that preserves orientation, lengths

and angles. In the most compressed form it is deﬁned as a 2 × 2 matrix R:

R =



cos θ − sin θ

sin θ cos θ



for some angle θ. In this 2 × 2 form, rotation is applied to an Euclidean point, and the

result is also in Euclidean form, and is the position of the initial point rotated by the angle

θ around the origin. Another more versatile form of the rotation matrix is

R =





cos θ − sin θ 0

sin θ cos θ 0

0 0 1





which is applied to a homogeneous point, and returns a rotated homogeneous point.

Deﬁnition 2.3.3. Scaling is a transformation that preserves orientation and angles. It

takes a form of a non-zero scaling factor s, which can be applied to both homogeneous

and inhomogeneous points.

2. MATHEMATICAL CONCEPTS 21

Rather than appearing separately, translation, rotation and scaling are often combined

to create more complex transformations. For example, a combination of rotation and

translation is called rigid or Euclidean transformation, and is of the form



cos θ − sin θ t

sin θ cos θ t



A combination of all three basic transformations (translation, rotation and scaling) is

referred to as similarity transform is deﬁned as



s cos θ −s sin θ st

s sin θ s cos θ st



or more generally as



a −b t

b a t



Deﬁnition 2.3.4. An aﬃne transformation is a type of transformation that preserves

parallel lines. It takes the form of any 2 × 3 matrix





Deﬁnition 2.3.5. A homography, also known as projective transformation or perspective

transform is a transformation of a two-dimensional image I to another two-dimensional

image I

(referred to as the anamorphic image), such that all straight lines in I are pre-

served in I

. It is represented as a 3 × 3 matrix









Lemma 2.3.6. A homography H produces the same transformation as a homography λH

for any non-zero λ.

Proof. Let p = (x, y)

be a point in 2D, and H be a homography matrix deﬁned as:

H =









2. MATHEMATICAL CONCEPTS 22

By Deﬁnition 2.2.1, (x, y, 1)

is a homogeneous representation of p. Let point q be the

result of the homography transformation H on the homogeneous point p. Then

q = Hp =





x + h

y + h

x + h

y + h

x + h

y + h











x+h

y+h

x+h

y+h

x+h

y+h

x+h

y+h







The last step of the derivation of q is valid because a homogeneous point remains un-

changed when multiplied by a scalar.

Let H

be another homography deﬁned as H

= λH for some non-zero λ. Let q

be the

result of the homography H

on p. Then

= H

p =







λ (h

x + h

y + h

)

λ (h

x + h

y + h

)

λ (h

x + h

y + h

)













λ(h

x+h

y+h

)

λ(h

x+h

y+h

)

λ(h

x+h

y+h

)

λ(h

x+h

y+h

)













x+h

y+h

x+h

y+h

x+h

y+h

x+h

y+h







Thanks to the property of scalar invariance of the homogeneous representation, the two

transformations H and H

= λH are the same.

By Lemma 2.3.6, projective transformation matrices are invariant to scale, and therefore

have eight degrees of freedom. This fact is of particular signiﬁcance as we introduce a

method for homography estimation that uses this property in Section 3.2.

2.3.2 Warping

Deﬁnitions 2.3.1 through 2.3.5 are diﬀerent types of two-dimensional transformations. The

process of applying a transformation, or a combination of transformations to an image is

called warping. We distinguish two basic types of warping: forward and backward, which

diﬀer in that forward warping directly transforms the original image to the ﬁnal image,

while backward warping ﬁlls in each pixel of the ﬁnal image with the corresponding color

from the original image.

2. MATHEMATICAL CONCEPTS 23

Deﬁnition 2.3.7 (Forward Warping). Let t be a transformation function that maps pixels

from image I to another image I

. An image can be forward warped by copying the color

at each pixel at point p ∈ I to a point p

∈ I

, where p

= t(p).

Deﬁnition 2.3.8 (Backward Warping). Let t be a transformation function that maps

pixels from image I to another image I

, and t

−1

be the inverse warping that maps pixels

from I

to I. An image can be backward warped by iterating through points p

∈ I

, and

“coloring” it with the color at the point p ∈ I, where p = t

−1

In practice, warping is often improved by some methods of interpolation that take into

account a cluster of pixels to determine the best color for each pixel on the ﬁnal image,

rather than copying a single pixel from the original image. Interpolation is designed to

create smoother and more accurate color transitions, and further minimizes the distortions

generated by warping. In this project, we use a warping method provided by the OpenCV

library which uses bilinear interpolation and nearest-neighbor interpolation [15]. These

interpolation methods determine the color value at each pixel by averaging over the pixels

around it.

Algorithmic Methods

This chapter deals with the techniques and methods used in the implementation of the

automated anamorphic projection program that we develop in this project. We introduce

the linear least squares minimization method, which is a crucial tool for estimating the

projective transformations of various surfaces in our anamorphic model, which we discuss

in more detail in Chapter 4. We also explain the main feature detection algorithms such

as Harris Corner Detection and Chessboard Corner Detection, which are used to facilitate

the process of computing the transformations.

3.1 Least Squares Minimization Method

The least squares method is a technique to estimate the best possible solution to a system

of equations, which is especially useful when the exact solution to that system does not

exist. This approach is designed to solve overdetermined systems, which means that there

are more equations than unknowns. This estimate is obtained by searching for a solution

that minimizes the error, deﬁned as the sum of squared diﬀerences of the result of the

estimated solution and the desired result. One of the most prevalent uses of this method

3. ALGORITHMIC METHODS 25

is in ﬁnding the line of best ﬁt to a set of data points, such that the sum of the squared

vertical distances from the line to each data point is minimized. The least squares model

can be used to predict the outcomes of the points not included in the ﬁnite input data.

For example, the line of best ﬁt provides a prediction of the outcomes for all points in

the domain, and not just the input data points. Because of its application as an error

minimization method, the least squares estimation is used in many optimization problems

in practical experiments to minimize the errors resulting from inaccuracies of a practical

setup. For the same reason, it is of great signiﬁcance to this project as it is the main

technique to estimate projective transformations between the computer screen image,

camera image, and the projective screen.

We distinguish two types of least squares methods: linear and non-linear least squares.

The linear least squares is an approach for ﬁnding the model that best describes a set of

input and output data, in which for pair of corresponding input and output data points,

the model can be expressed as a linear function of the parameters. Notice that the linear

least squares model is not necessarily a linear function in terms of the data points. For

example f(x) = ax

+ b log(x) is a valid model because f (x

) = ax

+ b log(x

) is linear

in terms of the coeﬃcients a and b for all data points x

. On the contrary, the non-linear

model is not linear in terms of the coeﬃcients, although some non-linear models can

be transformed into linear models, for example the non-linear model g(x) = e

can be

transformed into h(x) = ln (g(x)) = ax with the output data being ln(y) instead of just

y. We mentioned that the least squares solves overdetermined systems. This means that

for an assumed model with m coeﬃcients a

, a

, . . . a

we need n input data points, and

n corresponding output points, where n ≥ m.

This section deals with the linear least squares as it is the method used in this project.

We demonstrate the method of ﬁnding the least squares solution on a simple linear model,

and a more general model.

3. ALGORITHMIC METHODS 26

3.1.1 Linear Model

This is an example of the linear least squares on a linear regression model.

Let f : R −→ R be a linear function that models the data with input {x

, x

, . . . , x

} and

observed output {y

, y

, . . . , y

}, where n ≥ 2. The function f is of the form f(x) = mx+c

for some coeﬃcients m and c, where m 6= 0. In order for f to be a “good” model of the

data, we need to pick the values of m and c so that f(x

) ≈ y

for all 1 ≤ i ≤ n.

We deﬁne an error function E as E(m, c) =

i=1

(f(x

) − y

)

. To shorten the deriva-

tions and make it easier to read, we assume that all summations

in this subsection are

equivalent to

i=1

. The error function serves as a measure of how well f represents the

data. We need to ﬁnd m and c such that the value of E(m, c) is closest to 0. Because E is

always positive (E is a sum of squares), this statement is equivalent to ﬁnding m and c at

the minimum of E. In order to ﬁnd the minimum of E, we take the partial derivative of

E with respect to each of its arguments, and solve the system of equations in which each

derivative is equal to 0. The partial derivative of E with respect to m is

∂

∂m

E(m, c) =

∂

∂m

(mx

+ c − y

)

(mx

+ c − y

)

m +

c −

The partial derivative of E with respect to c is

∂

∂c

E(m, c) =

∂

∂c

(mx

+ c − y

)

2 (mx

+ c − y

)

m + (2n)c −

We set each derivative to zero and solve the simple system of linear equations in which

the ﬁrst column contains the coeﬃcients of m, and the second column corresponds to the

3. ALGORITHMIC METHODS 27

coeﬃcients of c, and the last column has the constant terms of the resulting equations:

We perform a simple row reduction to transform it to the reduced row echelon form:

∼





− (

)

0 n (

) − (

) (

)

∼







1 0

(

)

−

(

)(

)

(

)

−

(

)

0 n

−

(

)(

)

−

(

)

(

)

(

)

−

(

)







∼







1 0

(

)

−

(

)(

)

(

)

−

(

)

0 1

(

)

(

)

−

(

)(

)

(

)

−n

(

)







and obtain the optimal values for the coeﬃcients m, c:

m =

) − (

) (

)





− (

)

c =

(

)

(

) − (n

) (

)

n (

)

−





We substitute these values in the equation f (x) = mx + c to obtain the line of best ﬁt.

3.1.2 General Model

This section presents the linear least squares derivation on a general model.

Let f

, f

, . . . , f

be functions R −→ R and let f : R

−→ R be the model function,

deﬁned as

f(~x) = a

(~x) + a

(~x) + · · · + a

(~x),

where a

, a

, . . . , a

are the coeﬃcients of f. As we mentioned before, f does not need

to be a linear function on the variables ~x ∈ R

, but f must be linear in terms of its

3. ALGORITHMIC METHODS 28

coeﬃcients. This is important because in the process of ﬁnding the least squares solution,

the coeﬃcients of f become the unknowns, and the initial arguments of f take values

of the known input data, and form the coeﬃcients for the new unknowns. Let the input

data be { ~x

, ~x

, . . . , ~x

} and the corresponding observed output be {y

, y

, . . . , y

}, where

n ≥ m, and ~x

= (x

, x

, . . . , x

)

for all 0 ≤ i ≤ n. Let X be the matrix of the values

that the input data takes in function f , where row i of X contains the values of the input

data x

, deﬁned as:

X =







· · · X







where X

= f

(~x

). Notice that X

is independent of the coeﬃcients a

, a

, . . . , a

, which

satisﬁes the restriction of the linear least squares model.

Let ~a be the vector with all the coeﬃcients of f :

~a =













and let ~y be the vector containing the output data:

~y =













Then for each data point x

, the value of f is f (x

) =

· ~a, which we want to be equal

to y

, which means that we want

· ~a = y

. Our system can be represented as X~a = ~y,

expanded to:







· · · X































Because this is an overdetermined system, which means that the number of equations is

greater than the number of unknowns, the solution to this system might not exist. However,

3. ALGORITHMIC METHODS 29

we can still use the least squares method to estimate a solution for the coeﬃcients that

best approximates the input data to the observed output.

We deﬁne the error E function as:

E(a

, a

, . . . , a

) =

i=1

|f(~x

) − y

i=1



· ~a − y



= kX~a − ~yk

To ﬁnd the values of a

, a

, . . . , a

that minimize the error E, we ﬁnd the partial derivatives

of E with respect to the parameters a

and set them equal to zero. The system of linear

equations resulting from that operation is equivalent to X

X~a = X

~y. It remains to solve

the system for the unknown ~a. The product of the matrices X

and X is a square matrix

with size m, and X

~y is a vector of length m, so the number of equations is the same as

the number of unknowns, which means that we have an exact solution to the system. The

solution a

, a

, . . . , a

deﬁnes the model f that best ﬁts the data.

3.2 Projective Transformation

The relation between the anamorphic (distorted) image on a plane and the correct im-

age that the viewer observes is equivalent to that of an image projectively warped and

the original undistorted image, and so is simply a homography. This relation has in fact

been discussed in some previous research dealing with keystone correction [12]. Although

seemingly a separate problem, keystone correction involves the same plane-to-plane ho-

mographies, and the results of it can very well be adapted to our anamorphic problem.

The setup for this project consists of three surfaces detailed in Table 3.2.1.

Consider the transformation of an image from the projector screen to the projection

surface. Because the projection surface is a plane, this transformation is a homography.

Similarly, the transformation on the image from the projection surface to the camera

screen is also a homography because both of these surfaces are planar. If follows that the

3. ALGORITHMIC METHODS 30

Surface Explanation

camera image/screen image seen by the camera, also interpreted as the image seen

by the viewer

projector image/screen image projected by the projector; this is the image that the

computer gives to the projector, and not the projected image

on the projection surface

projection surface the actual projection on the screen (e.g. wall), with real-

life coordinates as opposed to the virtual coordinates of the

camera and projector image

Table 3.2.1: An explanation of the naming convention of the surfaces used in this project.

transformation from the projector screen to the camera screen is simply a composition of

the two homographies, and therefore is also a homography.

We diﬀerentiate between diﬀerent homographies by introducing the following notation:

is a homography mapping from surface A to surface B, where A, B ∈ {C, P, S}

with C, P, S corresponding to camera, projector, and projection surface respectively, rep-

resented on the diagram in Figure 3.2.1. For example, H

P C

is the homography mapping

each point in the projector image to a point on the camera image.

Figure 3.2.1: The homographies between the projector image, camera image, and projec-

tion surface.

3. ALGORITHMIC METHODS 31

3.2.1 Homography Estimation with Linear Least Squares

This subsection explains a method of homography estimation for a simple anamorphic

model using the least squares. While there are various methods of homography estimation,

for example the approach suggested in the Smarter Presentations model by Sukthankar et

al. [12], we present this method of estimating the homography as an illustrative example.

To simplify the notation, let H = H

P C

be the mapping from the projector to camera

plane, as shown in Figure 3.2.1. Its inverse, H

−1

, maps from the camera image back to

the projector image. The experimental design excludes the possibility of the last entry of

the matrix H being zero. This is because a homography with zero as the last entry maps

the point at the origin to a point at inﬁnity, as demonstrated in this example:

























This case is not applicable to this project because our controlled experiment ensures that

all relevant points, including the origin, are mapped to points that are ﬁnite in Euclidean

coordinates.

We formulate the problem as follows. Let {p

, p

, . . . p

n−1

} be a set of points in the

projector image, for some n ≥ 4. The reason for n to be at least 4 is explained later in this

section. Each point p

can be expressed in homogeneous coordinates as p

= (x

, y

, 1)

Let {p

, p

, . . . , p

n−1

} be the set of points in the camera image such that q

corresponds

to p

for all 0 ≤ i ≤ n − 1. Each point q

can be expressed in homogeneous coordinates as

= (X

, Y

, 1)

. Since the points in the projector image are transformed to points in the

camera image by projective transformation, any point p

is transformed into Hp

in the

camera image, where H is the homography mapping the projector to the camera image.

Without loss of generality, we assume that the last entry of H is 1, so H is of the form:

H =









3. ALGORITHMIC METHODS 32

for some unknown entries h

, h

, . . . , h

. It follows that for any point p

, we have







+ h

+ 1



















where the last step is true by the Deﬁnition 2.2.1 of the homogeneous point representation.

Our goal is to ﬁnd the values of h

, h

, . . . , h

such that Hp

≈ q

for all 0 ≤ i ≤ n − 1.

This can be expressed as:





















. (3.2.1)

We transform this system of non-linear equations into an equivalent system of linear

equations:





+ h









+ h

+ 1)

+ h

+ 1)





. (3.2.2)

For each point p

and the corresponding point q

, we have two constraints:

+ h

= X

+ h

+ 1)

+ h

= Y

+ h

+ 1) ,

which can be rewritten in terms of all variables h

, h

, . . . , h

+ y

+ h

+ 0h10 + 0h

+ 0h

− X

= X

+ 0h

+ x

+ y

+ h

− Y

= Y

Combining all equations resulting from all n point correspondences, we obtain the following

system of linear equations:







1 0 0 0 −X

−X

0 0 0 x

1 −Y

−Y

n−1

1 0 0 0 −X

n−1

−X

n−1

0 0 0 x

n−1

1 −Y

n−1

−Y

n−1

























n−1







3. ALGORITHMIC METHODS 33

Observe that n point correspondences give us 2n equations, and we have eight unknowns

, so we need n ≥ 4 so that the number of equations is at least the number of unknowns.

When n > 4, our system is overdetermined, which means that there is no exact solution

to it; in that case we use the least squares method. To shorten the notation of the system,

let A denote the matrix of coeﬃcients,

h denote the vector of unknowns, and

b denote the

vector of X

and Y

coordinates of points q

, so that our system of equations is A

h =

We deﬁne the error function E as:

E (h

, h

, . . . , h

) =

2n−1

i=0



h − b



= kA

h −

We need to ﬁnd

h that minimizes E, or in other words,

h at the minimum of function E. To

compute the minimum of E, we solve a system of equations in which the partial derivatives

of E are equal to zero. This is equivalent to solving A

h = A

b. Finally, we populate

the entries h

, h

, . . . , h

of the homography H with the entries of

h, and the last entry

of H with 1.

3.2.2 Homography Estimation with Homogeneous Linear Least Squares

The homography estimation method presented in the previous Subsection 3.2.1 is a con-

venient way to transform the nonlinear homography estimation problem into a linear one.

Recall that the goal of the previous estimation was to ﬁnd the entries h

, h

, . . . , h

the homography H that give the best solution to the system of nonlinear equations (3.2.1)

for all point correspondences. Finding the best solution to that system is equivalent to

minimizing the error function E deﬁned as:

E =



+ h

+ 1

− X





+ h

+ 1

− Y



3. ALGORITHMIC METHODS 34

Although this system is nonlinear, it can be easily transformed into an equivalent linear

system with the error function:

E =

+ h

− X

+ h

+ 1))

+ (h

+ h

− Y

+ h

+ 1))

This technique was used in the previous homography estimation method to create the

linear system, as shown in equation 3.2.2.

While this method oﬀers a simpliﬁcation to the nonlinear homography estimation prob-

lem and allows us to solve the problem with the linear least squares, it is not the only

way to approach this problem. This section introduces another method for estimating the

homography using the homogeneous linear least squares. We begin with formulating the

goal of this problem: we need to ﬁnd a homography H such that for every point p

on the

projector image and the corresponding point q

in the camera image, Hp

is approximately

equal to q

. An intuitive yet incorrect representation of the goal statement is Hp

≈ q

. It

is false because it fails to take into account the fact that in homogeneous representation,

represents the same point as λq

for any non-zero λ, so the best solution H might ap-

proximate Hp

to some multiple of q

. To address this problem we introduce a modiﬁed

equation Hp

≈ λ

, where λ

is an auxiliary scaling variable for every point q

. To ﬁnd

the best solution to this problem, we calculate H and λ

for 0 ≤ λ ≤ n − 1 that minimize

the error function

E =

kHp

− λ

By the linear least squares method, we set the partial derivatives of E to zero, and solve

the resulting system of linear equations to ﬁnd the best H and λ

, and then use the H to

compute the anamorph.

3. ALGORITHMIC METHODS 35

3.3 Feature Detection Methods

Feature detection is one of the most crucial tools to this project. Since our goal is to

create a system that automatically generates anamorphic images, we need a way to tell

the program about its surroundings, and more speciﬁcally about the surface that it needs

to project the anamorphic image on. A convenient way to gather information about the

scene is by using a camera to capture the necessary details of the scene. However, while

a person is able to easily recognize and diﬀerentiate between objects in the scene as soon

as he/she sees it, the image of the scene is nothing more than a collection of pixels to a

computer program. To help the computer understand visual signatures of the scene, we tell

it to ﬁnd some unique characteristics in the image based on the distribution and intensity

of the pixels. For example, analyzing an image in terms of the diﬀerence in intensity of the

neighboring pixels gives us information about the location of potential edges in the image.

In the context of this project, this is one way to detect the position of the projection

screen. Besides edges, other common easily detectable features include corners and blobs,

and these will prove especially useful in this project. In this section, we explore some

methods of feature and pattern detection, including the popular Harris Corner Detection,

and its improved version called Good Features to Track.

3.3.1 Harris Corner Detection

The Harris Corner Detection is one of the most common methods of feature detection.

It takes advantage of the observation that the change in intensity of the pixels around

a corner, relative to that corner is relatively large, so corners can be easily detected by

searching for pixels with the greatest variance in intensity of the pixels around it. This

procedure can be visualized as shifting a small window across an image, stopping at every

pixel and counting the sum of squared diﬀerences of the intensity of that pixel and each

pixel that is around it and inside the window.

3. ALGORITHMIC METHODS 36

We present the method based on the publication of Harris and Stephens in 1988 [5]. Let

I be an image. Without loss of generality, we assume that the image is in gray scale. Let

x,y

denote the intensity, or gray scale color, of the pixel at position (x, y) on the image.

Let E be the change in intensity resulting from a shift (u, v), deﬁned as:

u,v

x,y

x+u,y+v

− I

x,y

where w

x,y

is the window with unity in the desired region, and zero elsewhere.

A corner occurs when the change E is large in two dimensions, and is classiﬁed as a

relevant feature if the value of E at that corner is above some threshold value.

3.3.2 Good Features To Track

The Good Features to Track algorithm was developed by Shi and Tomasi [14] as an

improvement to the Harris corner detection, and a tool to facilitate feature-based vision

systems that require excellent tracking of features from frame to frame. This algorithm is

an improvement from the earlier feature detection methods in that it utilizes new tracking

algorithms [14,15].

Results

This chapter describes the implementation and results of diﬀerent models of perspective

planar anamorphosis. Speciﬁcally, the chapter presents the experimental setup, procedures,

and limitations for each model as well as explains the algorithms and main functions

utilized in the implementations. This chapter begins with a simple planar model that

serves as a platform for implementing more complex types of anamorphosis. Due to the

iterative nature of the project, later sections integrate functions and computations derived

in preceding sections as appropriate. Unless otherwise indicated, these functions serve the

same purpose as they ﬁrst appear in the chapter.

4.1 Basic Planar Model

The goal of this section is to build a program that generates an anamorphic image on a

planar surface. The only equipment needed is a projector and camera connected to a com-

puter, and a blank ﬂat surface. The setup for this model consists of three surfaces: camera

image, projector image, projection surface, deﬁned earlier in Table 3.2.1. For simplicity

purposes, we make several assumptions before the experiment. First, we assume that the

4. RESULTS 38

camera is the viewer in order to avoid the necessity to deﬁne the position of the viewer,

and to make it easier to record the image seen by the viewer. Second, we assume that

the camera and projector are both pointed at the projection surface (although preferably

from diﬀerent locations) so that the projector image is within the projection surface and

the camera’s scope encompasses the entire projection surface. It is also advised to keep

dim lighting because it makes the projection more distinguishable from the background

of the projection surface.

In order to maintain the consistency of the terms used in this chapter, we also introduce

the following naming convention:

Name Explanation

pre-anamorph the original image that will be used for warping

anamorph or

anamorphic

image

the warped image that looks askew before projecting, but

appears “straight” when viewed by the viewer (or the camera)

Table 4.1.1: An explanation of the naming convention for the anamorphic and pre-

anamorphic image.

The main task in generating the anamorphic image is ﬁnding the relevant homographies

that, when applied to the pre-anamorph, produce an anamorph that appears correctly in

the camera image. The algorithm for generating the anamorphic image is outlined in the

following four steps:

1. Project a pattern.

2. Find at least four common points in the projector and camera image.

3. Estimate homography H

P C

using the least squares method.

4. Use H

= H

−1

P C

to warp the original image.

Figure 4.1.1 provides a conceptual diagram illustrating this algorithm. In step 1. of

the algorithm, we project a simple rectangular pattern (Figure 4.1.1a), and capture the

4. RESULTS 39

projection with the camera (Figure 4.1.1b). We ﬁnd the homography H

P C

using the point

correspondences in the projector and camera image, and use H

P C

to create the anamorphic

image. Figure 4.1.2 illustrates the steps involved in projecting the anamorphic image and

verifying that it is seen correctly by the camera.

Figure 4.1.1: Steps involved in ﬁnding the homography H

P C

: (a) project a rectangular

pattern (b) Detect the pattern in the camera image

Figure 4.1.2: Verify that the anamorph is seen correctly by the camera: (a) project the

anamorph (b) The image (white rectangle) is seen correctly in the camera image.

The result of the algorithm for generating the anamorphic image is shown in the screen

shots and camera captures in Figure 4.1.3. The camera captures of the projection are taken

with the long exposure technique, which takes multiple pictures from the same position and

averages over them to reduce the blinking eﬀect of the projector. A detailed explanation

4. RESULTS 40

(a) Camera image with detected projection

corners; this corresponds to Fig. 4.1.1b.

(b) Pre-anamorph (original unwarped im-

age); target image.

sponds to Fig. 4.1.2a.

(d) Anamorph seen correctly by the camera;

this corresponds to Fig. 4.1.2b.

Figure 4.1.3: Screen shots and camera captures of the program

of each step of the algorithm, and a discussion of the challenges and limitations of this

model are provided in the following subsections.

4.1.1 Pattern Detection

A simple white rectangle is projected on the projection surface, and captured by the

camera. Next, the corners of the projected rectangle are detected in the camera image

using the Good Features to Track algorithm, described in Section 3.3.2. This algorithm

is convenient in that it allows us to specify the number of corners to be detected, and

returns that number of the “best” corners. The four corners of the rectangular projection

are correctly detected and marked with blue circles in the camera image as shown in 4.1.3a.

4. RESULTS 41

4.1.2 Homography Estimation

The corners of the projection detected in the previous subsection (4.1.1) are arranged in

the clockwise order starting from the top-left corner. The ordered corners from the camera

image are then paired up with the corners of initial pattern in the projector image, which

in this case are simply the outermost corners of the projector image, arranged in the same

clockwise order. We use the homography estimation method available in the OpenCV

Python library to obtain the best homography H

P C

relating the projector image to the

camera image. The OpenCV estimation function gives the same result as the homography

estimation method described in Section 3.2.1.

4.1.3 Generating the Anamorphic Image

The ultimate goal of this model is to obtain the undistorted original image in the camera.

The question is: What should we project to obtain correct the projective distortion of the

camera? The answer to this question becomes clear if we reverse our perspective. So far,

we have considered the transformation of images from the projector image to the camera

image. However, in this problem, the camera image is the given factor (we know that

the camera should obtain the undistorted original image), and the projector image is the

unknown that we need to solve for.

It follows that the anamorphic image (which will be projected) can be generated by

applying forward warping on the original image (Figure 4.1.3b), as it is exactly the target

image that we want to receive in the camera. For every pixel in the original image, we

apply H

to its homogeneous position (x, y, 1) to obtain a new position (x

, y

, ω) in the

projector image, and copy the pixel to that new location. The result is an anamorphic

image that looks distorted in the projector (computer) screen (Figure 4.1.3c), but appears

correct once projected and captured in the camera image (Figure 4.1.3d).

4. RESULTS 42

One technical issue that we encounter in this step of the algorithm as a result of the

setup of this experiment is that some parts of the camera image correspond to the area

outside of the projector image. For example, if we apply the homography H

directly

to the original image (pre-anamorph), then some parts of the resulting anamorphic image

will be outside of the projector image. Instead we must choose a target area within the

camera view that lies inside the projection area.

Figure 4.1.4 illustrates a situation in which a point in the camera image is transformed

to a point beyond the projector screen. The gray rectangle in the left (a) is the camera

image, and the white quadrilateral shape inside the rectangle is the projection area. This

quadrilateral projection area in the camera image corresponds to the projector image,

represented as the white rectangle on the right (b). A point p lies outside of the projection

area in the camera image (a). When the entire camera image is warped to the projector

image, the point p is transformed to a point p

in the projector image. Because p lies

outside of the projection area, the point p

must lie outside of the projector image.

Figure 4.1.4: Diagram of the scenario in which the points in the camera image are mapped

outside of the projector image: (a) camera image (b) projector image

A temporary solution to this problem is to reduce the size of the original image by

some ﬁxed factor, and place it in the center of the potential (target) camera image, which

4. RESULTS 43

is some estimate/guess of where the projection lands in the camera image. If the scale

factor is small enough, this solution would work for most projector-camera arrangements.

However, this method does not guarantee that the anamorph ﬁts inside the projector

image for all possible projector and camera positions, and might signiﬁcantly decrease the

size of the resulting anamorphic image. We will describe a better solution to this problem

in the Improved Model discussed in the next section (4.2).

4.1.4 Challenges and Limitations

As explained in Subsection 4.1.3, the current algorithm for generating the anamorphic

image might produce an image that does not ﬁt inside the projector image. An example

of this is shown in Figure 4.1.5.

(a)

(b)

Figure 4.1.5: (a) Anamorphic image not ﬁtting inside the projector image. (b) When the

anamorphic image in (a) is projected, the camera receives an incomplete image.

Figure 4.1.6 shows an example of incorrectly detected projection corners, which pre-

vented the program from generating a correct anamorphic image. This error might be

caused by poor lighting of the system, which aﬀects the contrast between the projection

and the projection surface, making it less detectable by the corner detection function. In

some situations, the projection corners are not detected due to the position of the cam-

era, which might be at an angle relative to the projection such that some parts of the

projection do not appear clearly in the camera image.

4. RESULTS 44

Figure 4.1.6: Incorrectly detected projection corners.

4.2 Improved Planar Model

The Basic Model described in the previous section (4.1) presents a simple method for

generating an anamorphic image based solely on the homography between the projector

and camera image. While eﬀective and suﬃcient in the simple scenario of our project,

the said method has three major limitations. First, it neglects the relationship between

the physical projection surface to the virtual projector and camera images. Although this

missing relationship does not aﬀect the process of generating the anamorphic images when

using a camera as the viewer, it is necessary for potential future directions of this project.

In particular, the homographies mapping to the projection surface are crucial if we want to

generate an anamorphosis for a real person as the viewer instead of the camera, and allow

us to easily adapt this system to create a keystone correction model. Second, the previous

method also fails to reliably eliminate the case of the anamorphic image not ﬁtting inside

the projector image. Lastly, the pattern detection method described in Subsection 4.1.1 is

deﬁcient because it supplies only four point correspondences, which is the minimum num-

ber of correspondences required to estimate the homography H

P C

. When the conditions

such as lighting, and projector and camera angle are unfavorable, this limitation hinders

4. RESULTS 45

the process of homography estimation because the error from detecting each of the four

corners of the rectangular pattern projected has a signiﬁcant eﬀect on the accuracy of the

resulting homography. This section uses a modiﬁed set-up to incorporate the projection

surface, proposes a more complex pattern detection algorithm, and introduces a method

for repositioning of the pre-anamorph to guarantee that it ﬁts inside the projector image

after warping.

(a) (b)

(c)

(d)

Figure 4.2.1: Screenshots of the program: (a) Detected chessboard corners. (b) Prepared

pre-anamorph. (c) Anamorphic image after warping. (d) Anamorphic image seen correctly

by the camera.

The modiﬁed algorithm is outlined below:

1. Project a chessboard pattern.

2. Find the inner corners of the chessboard in the projector and camera image.

4. RESULTS 46

3. Estimate the homography H

P C

using the least squares method.

4. Prepare (resize and reposition) the pre-anamorph.

5. Generate the anamorph by warping the pre-anamorph with the homography

= H

−1

P C

The result of this algorithm is compiled in Figure 4.2.1 and explained in detail later in

this section.

4.2.1 Modiﬁed Set-up

We modify our setup to make it possible to ﬁnd the homographies mapping to the projec-

tion surface. The Basic Model in Section 4.1 does not compute any homography involving

the projection surface, so there is no need for the projection surface to have any detectable

points. In fact, as seen in some camera captures of that model, for example Figures 4.1.3a

and 4.1.3d, the projection surface is simply a large blank planar surface, and it is desirable

that the the surface does not have any distinguishable features as they could be mistaken

by the program for the corners of the projected rectangle that we need to detect. However,

the new setup introduced in this section requires that the projection surface has some de-

tectable features to facilitate the process of ﬁnding the homography mapping to it. We add

a rectangular projection surface, on which we project images with the projector. We use

a large rectangular piece of paper attached to a wall. The entire projection surface is then

captured by the camera, so that the corners of the projection surface can be detected in

the camera image. To ﬁnd the homography H

from the projection surface to the camera

image, we use the same homography estimation method as the one used to estimate the

projector-camera homography H

P C

4. RESULTS 47

4.2.2 Detecting the Projection Screen

We use the same good features to track algorithm to detect the corners of the projec-

tion screen. Unlike the projection (light from the projector), the screen is a solid object

and its color is appears stable in the camera, which means that the diﬀerence in color

intensity between the projection screen and the background is more easily detectable than

that between the projection and the background. Therefore the Good Features to Track

algorithm, described in Section 3.3.2, is relevant as it provides a convenient way to de-

tect exactly four corners of the projection surface. The detected corners are presented in

Figure 4.2.2.

Figure 4.2.2: Detected projection screen corners marked with blue dots.

These corners in the camera image correspond to the corners of the projection surface in

the real world. Using these four point correspondences, we can ﬁnd the homography H

by using the estimation method described in Subsection 3.2.1, where the real-world corners

of the projection surface assume the role of the projector points in the estimation method,

and the detected corners of the surface in the camera image are the output points..

4. RESULTS 48

4.2.3 Homography Estimation with Chessboard Corner Detection

The Basic Model described in Section 4.1 uses a simple corner detection function to ﬁnd

the corners of a rectangular shape projected on a wall, as seen in the camera image

(Figure 4.1.3a), and matches these corners with the corners of the original shape that was

projected, as seen on the computer screen. These four corner correspondences are then

used to ﬁnd the homography H

P C

. This method requires a speciﬁc light setting, and might

not produce the desired results when the light is too bright as the projection might not

be clearly visible in the camera image.

(a)

(b)

Figure 4.2.3: (a) Original chessboard corners in the projector image. (b) Detected chess-

board corners in the camera image.

This subsection introduces an alternative way to obtain points in the projector and cam-

era image that provide a better estimation of the homography H

P C

. The major amendment

is in the pattern projected. Instead of the simple rectangular pattern that contains only

four detectable features, we project a 6×9 chessboard pattern, presented in Figure 4.2.3a,

with 5 · 8 = 40 detectable inner corners. We choose this pattern because it is easily dis-

tinguishable from the projection screen. Notice that if we repeat the pattern detection

process from the Basic Model (Section 4.1), the algorithm used to detect the projection

4. RESULTS 49

corners might not be able to distinguish between the corners of the projection and the

projection surface. The chessboard provides an eﬃcient solution because it can be easily

detected regardless of other detectable objects surrounding it in the camera image, as

shown in Figure 4.2.3b.

4.2.4 Repositioning and Resizing the Pre-anamorph

A major limitation of the Basic Model is the fact that the anamorph may not ﬁt entirely

within the projector image after warping. Recall that this phenomenon occurs because the

entire projector image is mapped by H

P C

into an area inside the camera image. It follows

that the inverse homography H

maps the points within that projection area (inside the

camera image) back to their positions in the projector image. However, the points in the

camera image that lie outside of the projection area are mapped to new positions that do

not ﬁt inside the projector image, as illustrated in Figure 4.1.4.

We propose a method for resizing and repositioning the pre-anamorph before warping

it to ensure that the resulting anamorph ﬁts entirely inside the projector image and is of

the largest size possible, described in the following steps:

1. Identify the four corners of the quadrilateral region of the projection area inside the

camera image by applying H

P C

to each of the four outer corners of the projection

image. The detected projection is shown in Figure 4.2.4a.

2. Find the largest rectangle that ﬁts inside the projection region in the camera image,

as shown in Figure 4.2.4b. We explain this in more detail below.

3. Resize the original image so that it ﬁts inside that rectangle, and place it in a blank

(uniformly colored) camera image in a position that is aligned with the position of

that rectangle (for example by aligning the top-left corner of the pre-anamorph with

the top-left corner of that rectangle), as shown in Figure 4.2.4c.

4. RESULTS 50

(a) Corners of the projection area in the cam-

era image.

(b) Maximal rectangle with ﬁxed aspect ratio

that ﬁts inside the projection area.

Figure 4.2.4: Screenshots of the program

After accomplishing these steps, the pre-anamorph is ready to be warped and is ensured

to land inside the projector image.

It remains to explain the method for ﬁnding the largest rectangle inside a quadrilateral

ﬁgure. Our method iterates through the pixels in the camera image and searches for the

largest rectangle that ﬁts inside the projection area. We improve the run-time of the

algorithm by iterating only through the pixels that lie inside the projection area, before

searching for the best rectangle, and we stop the iteration once the remaining area in the

4. RESULTS 51

camera image is smaller than the current largest rectangle. Additionally, we require that

the rectangle has the same aspect ratio as the original image.

4.2.5 Challenges and Limitations

One minor issue we encounter when testing this model is that for some positions of the

camera, the chessboard pattern is not detected correctly. One possible explanation is that

the camera is at an angle (relative to the projection screen) such that the chessboard

pattern appears too small, which causes the chessboard corners to be too crowded to be

detected correctly. In addition, the projector light is not stable, which reduces the quality of

the camera picture and makes it more diﬃcult to accurately detect the chessboard corners.

However, none of the tests performed have shown a noticeable deterioration in the quality

of the anamorphic image. Because the number of chessboard corners is signiﬁcantly larger

than four, which is the minimum number of points required to establish a homography, the

inaccuracy in detecting some of them is less likely to signiﬁcantly aﬀect the ﬁnal result.

Figure 4.2.5: A scenario in which the program fails to detect the chessboard corners cor-

rectly.

4. RESULTS 52

4.3 Two-view Anamorphosis

By the deﬁnition of anamorphosis, there is only one perspective, or rather one ray, such

that when a viewer looks at the anamorphic image from any point on that ray, the anamor-

phic image appears as the original unwarped image to the viewer. However, in practice

there might be a number of viewers looking at the anamorphic image from several perspec-

tives. In this section we address this issue and test a method for ﬁnding a transformation

that minimizes the distortions for the viewers. Once again, we use cameras to represent

viewers, and assume that the camera captures are what the viewers observe. Our goal is

to produce a homography

H such that when

H is applied to an image, and the resulting

anamorphic image is projected onto the projection surface, each camera capture of the

projected image is “as close as possible” to the original unwarped image. Strictly follow-

ing the deﬁnition of anamorphosis, the image transformed with that homography is not

an anamorphosis for any of the cameras, because an anamorphosis can satisfy only one

perspective. Rather than generating an image that is an anamorphosis for all the view-

ers, which is theoretically impossible, we aim to generate an image that is “as close as

possible” to the anamorphosis for each viewer. Although the result of this program is not

an anamorphic image, for consistency, we continue to use the terms “pre-anamorph” and

“anamorph” to refer to the unwarped image and the ﬁnal warped image that is projected,

respectively.

The ﬁrst question we need to answer is: how do we deﬁne “closeness” between the

desired image and the actual image we obtain? In other words, what is the error? One

way to deﬁne it is in terms of distance, which in this scenario means the distance between

the same pixel in the two images in the x and y direction. We take the sum of the squared

distances at each pixel as a measure of the error between the obtained and target image.

Based on this deﬁnition, our reformulated goal is to ﬁnd a homography such that when

4. RESULTS 53

it is applied to an image, and the image is projected on the projection surface, and then

captured by the two cameras, the sum of the errors between the captures and the desired

images is minimized. This problem is diﬀerent from the previous two models in Sections

4.1 and 4.2 in that we need to minimize the error for both camera images, which makes

the estimation of the optimal homography to warp the original image to the projector

image signiﬁcantly more complex.

The inclusion of the second camera requires that we introduce some changes to the

program. We present the updated outline of the algorithm to generate an optimal image

for two viewers:

1. Project a chessboard pattern.

2. Find the inner corners of the chessboard in Camera 1 and Camera 2 images.

3. Estimate the projector-camera homography for each camera, using the same method

as in the Improved Model in Section 4.2.

4. Choose a target chessboard pattern for each camera.

5. Estimate the homography

H and its inverse

−1

that map between the original im-

age and the projector image, and minimize the diﬀerence in the observed chessboard

corners and the target chessboard corners for each camera. The new homography

notation is explained in the following Subsection 4.3.1.

6. Prepare (resize and reposition) the pre-anamorph.

7. Generate the anamorph by warping the pre-anamorph with the homography

−1

4.3.1 Two-view Setup

The equipment needed for this setup is a projector, and projection surface, and two cam-

eras. As in the previous models, we require that the projector and the cameras are directed

towards the projection screen, and that the images projected land inside the projection

4. RESULTS 54

screen, and the cameras capture the entire projection screen. This setup is visualized in

Figure 4.3.1.

Figure 4.3.1: The new setup: (a) Camera 1 (b) Projector (c) Camera 2 (d) Projection

surface.

Along with the changed setup, we also introduce a new notation of the homographies for

this particular two view model, as presented in Figure 4.3.2. We denote the homography

from the projector image to the ﬁrst camera image as A, and the homography from the

projector image to the second camera image as B. A and B correspond to the projector-

camera homography H

P C

from the one view models discussed earlier. We introduce a

new homography

H from the projector image to the original image. It follows that the

inverse homography

−1

maps from the original image to the projector image, and that

−1

and B

−1

map from the original image to the ﬁrst and second camera image

respectively. Recall that the previous one-view models use the homography H

to warp

the pre-anamorph into the anamorph. This method stems from the valid assumption that

the pre-anamorph is the target camera image, and the anamorph is the projector image,

so the anamorph can be created by warping the target camera image into the projector

image. However, this approach is not applicable to this model because the homography

used to warp the pre-anamorph into the anamorph is neither A

−1

(the camera-projector

homography for camera 1) nor B

−1

(the camera-projector homography for camera 1). In

4. RESULTS 55

Figure 4.3.2: The new setup and notation for the homographies.

fact, if either A

−1

or B

−1

is used to create the anamorph, the result will be optimized for

only one of the cameras. Therefore, this model requires a new homography

H to optimize

the projected image for both camera views. The primary task in this model is to estimate

the best homography

H as we need it to warp the original image (pre-anamorph) into the

optimal anamorphic image, which is then projected.

4.3.2 Target Camera Images

The steps 1 through 3 outlined at the beginning of this section give an estimate of the

homographies A and B using the estimation method described in the previous one-view

models. It remains to determine the homography

H. Before we introduce the new method

for homography estimation, we reiterate the approach used in the previous models to

explain why it does not apply in the current two-view model, which sheds some light on

the modiﬁcations we need to make to adapt the previous method to this scenario.

In the Improved Model approach, we project a chessboard pattern and detect the trans-

formed chessboard corners in the camera image, as explained in Subsection 4.2.3. We

then investigate the relationship between the original chessboard corners (in the projec-

tor image) and those detected in the camera image, and ﬁnd the best homography H

P C

that transforms the original chessboard points to the camera points. The reason why this

4. RESULTS 56

method can only be used to estimate the homographies A and B, but cannot be used

to estimate the homography

H is that the previous approach considers the relationship

of points between the projector and camera image, while

H relates points between the

projector image and the original image (or the pre-anamorph).

To address this issue, we introduce a new estimation approach, which allows us to solve

for the best homographies AH

−1

and BH

−1

between the original image and the camera

image. Because A and B are already known to us from the earlier steps of the program,

we eﬀectively solve for H

−1

. To do this, we need a set of points in the original image,

and a corresponding set of points in each camera image. The points on the original image

are simply the original chessboard corners. The target points on each camera image are

the same as the original chessboard points, up to scale and translation. This follows from

the reasoning that when the original image is the pre-anamorph (i.e. the picture that we

use to generate the ﬁnal anamorphic image), our goal is for the cameras to see the same

picture as the original image, up to scale and translation.

To account for the location of the actual projection in each camera image, we ﬁnd

the new chessboard corners by performing a least squares minimization on the detected

chessboard corners. In addition, we impose the following constraints on the new chessboard

pattern:

1. The number of rows and columns in the new pattern is the same as that of the

original chessboard pattern.

2. The chessboard corners are equally spaced.

3. The grid lines of the new pattern are parallel to the x and y axes.

The result of this method is presented in Figure 4.3.3

4. RESULTS 57

(a) New chessboard pattern for Camera 1

shown with red dots.

(b) New chessboard pattern for Camera 2

shown with red dots.

Figure 4.3.3: New chessboard patterns.

4.3.3 Homography Estimation

Due to the changes in the setup, the homography estimation method used in the previ-

ous models no longer applies. Therefore, we introduce a homography estimation method

designed speciﬁcally for this model. While this technique bears some similarities to the

previous method in that we continue to use the least squares minimization to ﬁnd the

optimal homography

H, the function that we minimize on is signiﬁcantly diﬀerent. This

subsection provides a detailed explanation of the new homography estimation using the

homogeneous least squares for the two view model.

Let p

, p

, . . . , p

n−1

be the chessboard corners in the original image. Let v

, v

, . . . , v

n−1

and w

, w

, . . . , w

n−1

be the target chessboard corners in Camera 1 and Camera 2 respec-

tively. The homography that maps points from the original image to a camera image is

−1

for Camera 1 and B

−1

for Camera 2, where the entries of

−1

are the unknowns.

Let h

, h

, . . . , h

be the entries of

−1

so that

−1









4. RESULTS 58

Without loss of generality, we assume that h

= 1. By the homogeneous least squares

described in Subsection 3.2.2, the error function E can be deﬁned as:

E(h

, h

, . . . , h

, λ

, . . . , λ

n−1

, µ

, . . . , µ

n−1

) =

n−1

k=0





−1

− λ





n−1

k=0





−1

− µ





for some scaling variables λ

corresponding to each target point v

in Camera 1, and µ

corresponding to each target point w

in Camera 2. The scaling factors account for the

fact that any multiple of a homogeneous point corresponds to the same point, and so

taking the diﬀerence between A

−1

and a multiple of v

, and the diﬀerence between

−1

and a multiple of w

gives a better estimate of the actual distance between these

pairs of points than, for example taking the diﬀerence of A

−1

and unscaled v

Observe that



−1

− λ



and



−1

− µ



are of similar structure, which

allows us to rewrite the error function in an equivalent but simpler form:

E(X

, X

, . . . , X

, σ

, . . . , σ

2n−1

) =

2n−1

k=0

− σ

where A

is A or B, X is

−1

, b

is p

k−n

, σ

is λ

or µ

k−n

, and c

is v

or w

k−n

, depending

on the value of k, as shown in Table 4.3.1.

X b

0 ≤ k ≤ n − 1 A

−1

n ≤ k ≤ 2n − 1 B

−1

k−n

Table 4.3.1: The meaning of new variables A

, X, b

, σ

and c

The goal of this section is to ﬁnd (h

, h

, . . . , h

, σ

, . . . , σ

2n−1

) that minimize the

error function. As explained in Section 3.1, we ﬁnd partial derivatives of E with respect to

each variable, and solve a system of linear equations in which each partial derivative is zero.

For simplicity, we perform the diﬀerentiation on the simple form of the error function. The

4. RESULTS 59

solution to that system of equations is the vector (X

, X

, . . . , X

, σ

, . . . , σ

2n−1

). In

particular, the values X

, X

, . . . , X

in the solution are the entries of the homography

matrix

−1

The partial derivatives take two diﬀerent forms depending on the variable with respect to

which we diﬀerentiate. We distinguish these two types of variables: X

where 0 ≤ r, s ≤ 2,

and σ

where 0 ≤ r ≤ 2n − 1.

The partial derivative of E with respect to X

for any r, s such that 0 ≤ r, s ≤ 2 is:

∂

∂X

2n−1

k=0

− σ

= 2

− σ

) · (A

) (4.3.1)

= 2

− σ

)

(4.3.2)

= 2

((A

)

− σ

)

) ((A

)

) (4.3.3)

= 2









i,j

)





− σ

)





((A

)

) (4.3.4)

= 2





i,j

)

−

)





The E

in lines (4.3.1) and (4.3.2) of the derivation above is deﬁned as a matrix with

one at the position rs, and zero everywhere else. For aesthetic purposes we skip the bounds

on the sums in the derivation above, and explain them here. The index iterator l results

from taking the dot product of A

− σ

and A

, each of which are vectors of

length 3, therefore 0 ≤ l ≤ 2. The index iterators i and j result from taking the l

entry

of the product of A

, X and b

. Since A

and X are 3 × 3 matrices, and b

is a vector

of length 3, it follows that 0 ≤ i, j ≤ 2. From (4.3.1) to (4.3.2), we use the fact that the

dot product of two vectors ~u and ~v is ~u · ~v =

. From (4.3.2) to (4.3.3) we use the

4. RESULTS 60

following derivation:

)

= (A

))



(0, . . . , (b

)

, . . . , 0)



, where (b

)

is the r

entry

= (A

)

From (4.3.3) to (4.3.4) we use these two properties:

(A~v)

, and

(AB~v)

(B~v)









We set this derivative to zero, and rearrange it so that the constant terms are on the

right-hand side. Because X

= 1, the term X

and its coeﬃcients are moved to the

right-hand side:





i,j

)

−

)





= 0

i,j

)

−

)

= 0

(i,j)6=(2,2)

)

−

)

. (4.3.5)

Converting the equation (4.3.5) back to the notation used in our initial setup of this

model, we have the following equation:

(i,j)6=(2,2)

)

+ B

)

−

)

−

)

= −

)

+ B

)

] .

4. RESULTS 61

The partial derivative with respect to σ

for any 0 ≤ r ≤ 2n − 1 is:

∂

∂σ

2n−1

k=0

− σ

∂

∂σ

− σ

= −2 (A

− σ

) · c

= −2

− σ

)

= −2









i,j

)





− σ

)





)

= −2





i,j

)

−

)





Notice that in the process of taking the derivative, the summation on k disappeared.

This is because

∂

∂σ

− σ

= 0 for all k 6= r. We explain the limits on the

summations above: l is the iterator in the dot product of A

− σ

and c

, both of

which are vectors of length 3, so 0 ≤ l ≤ 2; i and j are iterators in taking the l

entry of

the product of A

, X and b

, and since A

and X are 3 × 3 matrices and b

is a vector of

length 3, it follows that 0 ≤ i, j ≤ 2.

Again, we set the derivative to zero and bring the constant terms to the right-hand side:

−2





i,j

)

−

)





= 0

i,j

)

−

)

= 0

(i,j)6=(2,2)

)

−

)

. (4.3.6)

Converting to the notation used in our initial setup of this model, we obtain two equa-

tions depending on the value r. When 0 ≤ r ≤ n − 1, σ

= λ

, and the equation (4.3.6) is

(i,j)6=(2,2)

)

−

)

= −

)

4. RESULTS 62

When n ≤ r ≤ 2n − 1, σ

= µ

r−n

and the equation (4.3.6) is

(i,j)6=(2,2)

r−n

)

r−n

)

−

r−n

)

= −

r−n

)

r−n

)

We solve the resulting system of equations for the unknowns h

, h

, . . . , h

, λ

, . . . , λ

n−1

, µ

, . . . , µ

n−1

, where h

, h

, . . . , h

and 1 are the entries of the homog-

raphy

−1

4.3.4 Generating the Anamorphic Image

To generate the anamorphic image, we perform the repositioning and resizing steps de-

scribed in Subsection 4.2.4 in the Improved Model. We then apply the

−1

to the prepared

pre-anamorph, which lies on the original image, to transform it into the anamorph on the

projector image.

Our initial attempts did not provide expected results prompting us to identify some

potential shortcomings in our model. We tested the algorithm for homography estima-

tion using the homogeneous least squares method on speciﬁc parameters to verify if the

algorithm is correct. We compare the obtained test results with a parallel computation

in Mathematica. The results of the tests suggest that the implementation works for pa-

rameters that can be expressed with few decimal digits, but does not give the expected

solution for parameter with many decimal digits. This ﬁnding led us to consider the possi-

bility of encountering ill-conditioned matrices, which are matrices in which the diﬀerence

in between the smallest and largest entries is large, which causes the program to lose

some accuracy when performing Gaussian elimination on those numbers. Further tests

conﬁrmed this conjecture as the condition number was between 4 and 10, which classiﬁes

the coeﬃcient matrix in our model as ill-conditioned (a well-conditioned matrix has the

condition number approximately 1).

We attempted to address this problem by using the singular matrix decomposition.

Despite some improvements in accuracy when calculating the homography on the test

4. RESULTS 63

parameters, the problem of generating the anamorphic image remains unresolved. Due to

the time constraints, we hope to look further into this issue in future research.

Future Work

In this project we explore the mathematical concepts behind perspective planar anamor-

phosis, and develop models for generating the anamorphic images on planar surfaces for

single and two-view model using a simple projector-camera system. We design and imple-

ment a homography estimation method using homogeneous least squares for the two-view

model. However, we encounter some challenges that need to be addressed in the future.

Some improvements include developing more accurate feature detection techniques, and

solving the ill-conditioned matrices problem, which might improve the results of the two-

view model.

This project may serve as a basis for other research directions. One adaptation of the

ﬁndings of this work is in keystone correction, which will take advantage of the modiﬁed

setup in the improved model in Section 4.2. This project could be extended by including

a real-time tracking of the viewer’s position to create a dynamic anamorphosis system.

Another direction that could be taken is investigating anamorphosis for arbitrary surfaces,

which could be done for example with adaptive structured lighting techniques. The topics

studied in this project have a potential forother practical applications such as enhanc-

5. FUTURE WORK 65

ing viewers’ experience by embedding projected images in performances such as plays or

dance. To expand the scope of the applications of anamorphosis in performances, one

could develop a model of anamorphosis for more than two viewers, which can ultimately

be used to optimize the projection for a large audience. One way to address the large

audience scenario is to build a synchronized system of multiple projectors, which can po-

tentially generate more powerful and engaging images. A paper by Elodie presents another

interesting approach to the subject of anamorphosis by considering the Renaissance art

and investigating the way artists of that time dealt with perspective. [4] This approach

may shed some light on how humans perceive the world, and cast some doubt on whether

strictly following the perspective principles yields an image that is convincing to the hu-

man eye.

Bibliography

[1] R. E. Allardice, The Barycentric Calculus of M¨obius, Proceedings of the Edinburgh

Mathematical Society 10, 2–21.

[2] Alessandro Brazzini and Carlo Colombo, Computer vision for interactive skewed video

projection, Image Analysis and Processing–ICIAP 2005, 2005, pp. 139–146.

[3] Wilhelm Burger and Mark J. Burge, Digital image processing: an algorithmic intro-

duction using Java, Springer Science & Business Media, 2009.

[4] Elodie Fourquet, Learning about shadows from artists, Proceedings of the Sixth in-

ternational conference on Computational Aesthetics in Graphics, Visualization and

Imaging, 2010, pp. 107–114.

[5] Chris Harris and Mike Stephens, A combined corner and edge detector., Alvey vision

conference 15 (1988), 50.

[6] Richard Hartley and Andrew Zisserman, Multiple view geometry in computer vision,

Cambridge University Press, 2003.

[7] J. L. Hunt, BG Nickel, and Christian Gigault, Anamorphic images, American Journal

of Physics 68 (2000), no. 3, 232–237.

[8] Jean Fran¸cois Niceron, Perspective curieuse, JD Puis, 1992.

[9] Robert Ravnik, Borut Batagelj, Bojan Kverh, and Franc Solina, Dynamic Anamor-

phosis as a Special, Computer-Generated User Interface, Interacting with Computers

(2013), iwt027.

[10] David Eugene Smith and Mansﬁeld Merriman, History of modern mathematics, J.

Wiley & Sons, 1896.

[11] Franc Solina and Borut Batagelj, Dynamic anamorphosis (2007).

[12] Rahul Sukthankar, Smarter presentations: Exploiting homography in camera-projector

systems, Computer Vision, 2001. ICCV 2001. Proceedings. Eighth IEEE International

Conference 1 (2001), 247–253.

Bibliography 67

[13] Richard Szeliski, Computer vision: algorithms and applications, Springer Science &

Business Media, 2010.

[14] Carlo Tomasi and Jianbo Shi, Good features to track, CVPR94 600 (1994), 593–593.

[15] OpenCV documentation, http://docs.opencv.org/.